{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a5bfb7d8",
   "metadata": {},
   "source": [
    "# Evaluating your Elasticsearch LLM applications with Ragas\n",
    "\n",
    "## Installing the required packages\n",
    "\n",
    "You will first want to install the packages required for this application. These include the following:\n",
    "\n",
    "- The [Elasticsearch Python client ](https://www.elastic.co/docs/reference/elasticsearch/clients/python)will be helpful for authenticating and connecting to Elasticsearch and for vector search. \n",
    "\n",
    "- [Ragas](https://docs.ragas.io/en/stable/) is used for evaluating the quality of the LLM applications using standard metrics\n",
    "\n",
    "- The Hugging Face library [datasets](https://huggingface.co/docs/datasets/en/index) is used to create a robust evaluation dataset \n",
    "\n",
    "- [Langchain-OpenAI](https://python.langchain.com/api_reference/openai/llms/langchain_openai.llms.base.OpenAI.html), which is used both for generating answers to user questions and for  evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b21e3c09-f269-4c42-adc2-35f429d05ed1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m25.1.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m25.2\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install -q elasticsearch ragas datasets langchain-openai"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3ccc0c0",
   "metadata": {},
   "source": [
    "## Import modules\n",
    "\n",
    "After installing the required packages, import modules for handling environment variables, parsing JSON data, securely entering sensitive values, connecting to Elasticsearch, evaluating RAG applications and their metrics, creating evaluation datasets, and enabling chat model capabilities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4fd48912-9200-4fe9-8162-b0aefe697055",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import json\n",
    "from getpass import getpass\n",
    "from elasticsearch import Elasticsearch\n",
    "from ragas import evaluate\n",
    "from ragas.metrics import faithfulness, context_recall, context_precision\n",
    "from datasets import Dataset\n",
    "from langchain_openai import ChatOpenAI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "006ad4a6-9cf7-4506-82d2-992000f049fa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Host:  ········\n",
      "API Key:  ········\n"
     ]
    }
   ],
   "source": [
    "# Create an Elasticsearch client by securely prompting for the host URL and API key\n",
    "\n",
    "es = Elasticsearch(\n",
    "    getpass(\"Host: \"),\n",
    "    api_key=getpass(\"API Key: \"),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6e0cd5f5-5cad-4ad1-ab9d-4c260faeaff1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Name of the Elasticsearch index where book data is stored\n",
    "\n",
    "index_name = \"ragas-books\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4ea039c",
   "metadata": {},
   "source": [
    "Your model will need to be deployed first before the following code could run. To learn more about this be sure to check out our [documention on the subject](https://www.elastic.co/docs/explore-analyze/machine-learning/nlp/ml-nlp-deploy-model)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d02ee988-1b08-4c43-b39d-88c1a6bacf72",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert a text query into an embedding vector using e5.\n",
    "\n",
    "\n",
    "def embed_query(text: str):\n",
    "    res = es.ml.infer_trained_model(\n",
    "        model_id=\".multilingual-e5-small_linux-x86_64\",\n",
    "        body={\"docs\": [{\"text_field\": f\"query: {text}\"}]},\n",
    "    )\n",
    "\n",
    "    vec = res[\"inference_results\"][0][\"predicted_value\"]\n",
    "\n",
    "    return vec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "507e64cb-40a3-4657-9984-d9ab264014a7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deleted existing index 'ragas-books'\n",
      "Created index 'ragas-books'\n"
     ]
    }
   ],
   "source": [
    "# If the Elasticsearch index already exists, delete it to start fresh\n",
    "\n",
    "if es.indices.exists(index=index_name):\n",
    "    es.indices.delete(index=index_name)\n",
    "    print(f\"Deleted existing index '{index_name}'\")\n",
    "\n",
    "\n",
    "# Create a new index with a mapping that defines the fields and their types\n",
    "\n",
    "es.indices.create(\n",
    "    index=index_name,\n",
    "    body={\n",
    "        \"mappings\": {\n",
    "            \"properties\": {\n",
    "                \"book_title\": {\"type\": \"text\"},\n",
    "                \"author_name\": {\"type\": \"text\"},\n",
    "                \"book_description\": {\"type\": \"text\"},\n",
    "                \"rating_score\": {\"type\": \"float\"},\n",
    "                \"embedding\": {\n",
    "                    \"type\": \"dense_vector\",\n",
    "                    \"dims\": 384,\n",
    "                    \"index\": True,\n",
    "                    \"similarity\": \"cosine\",\n",
    "                },\n",
    "            }\n",
    "        }\n",
    "    },\n",
    ")\n",
    "print(f\"Created index '{index_name}'\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e300a8e0",
   "metadata": {},
   "source": [
    "If you encounter any problems running this in Colab or locally, it might be due to the dataset requiring a separate download. It can be found in the same folder as this example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ace8e9fe-c5b8-45c0-bbf7-f6b845fadd61",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Indexed 1: Lucky 7\n",
      "Indexed 2: Salvation Lost\n",
      "Indexed 3: Alien Warrior's Mate\n",
      "Indexed 4: On the Steel Breeze\n",
      "Indexed 5: Salvage Marines\n",
      "Indexed 6: Trade Secret\n",
      "Indexed 7: There Will Be Time\n",
      "Indexed 8: Only in Death\n",
      "Indexed 9: His To Claim\n",
      "Indexed 10: Savage Drift\n",
      "Indexed 11: Light of the Jedi\n",
      "Indexed 12: Mega Robo Bros\n",
      "Indexed 13: Transmetropolitan, Vol. 1: Back on the Street\n",
      "Indexed 14: The Queen of Traitors\n",
      "Indexed 15: The Island of Doctor Moreau\n",
      "Indexed 16: Human Nature\n",
      "Indexed 17: Legion\n",
      "Indexed 18: Wolfsbane\n",
      "Indexed 19: Lamb: The Gospel According to Biff, Christ's Childhood Pal\n",
      "Indexed 20: Our Pet\n",
      "Indexed 21: The Aylesford Skull\n",
      "Indexed 22: Ghosts of War\n",
      "Indexed 23: The Book of Time\n",
      "Indexed 24: Because It Is My Blood\n",
      "Indexed 25: The Annals of the Heechee\n"
     ]
    }
   ],
   "source": [
    "# Load the book data from a local JSON file into a Python list of dictionaries\n",
    "\n",
    "with open(\"books.json\") as f:\n",
    "    books = json.load(f)\n",
    "\n",
    "# Loop through each book record and index it into Elasticsearch\n",
    "\n",
    "for i, book in enumerate(books, 1):\n",
    "    try:\n",
    "        book[\"embedding\"] = embed_query(book[\"book_description\"])\n",
    "        es.index(index=index_name, document=book)\n",
    "        print(f\"Indexed {i}: {book['book_title']}\")\n",
    "    except Exception as e:\n",
    "        print(f\"Failed to index '{book.get('book_title', 'Unknown')}': {e}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b431670a-86d2-4226-a4bf-8b6aeaa54cab",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Runs a kNN search in Elasticsearch to find the most relevant books.\n",
    "\n",
    "\n",
    "def vector_search(query, top_k=3):\n",
    "    query_vector = embed_query(query)\n",
    "\n",
    "    body = {\n",
    "        \"knn\": {\n",
    "            \"field\": \"embedding\",\n",
    "            \"k\": top_k,\n",
    "            \"num_candidates\": 100,\n",
    "            \"query_vector\": query_vector,\n",
    "        },\n",
    "        \"_source\": [\"book_title\", \"author_name\", \"book_description\", \"rating_score\"],\n",
    "    }\n",
    "\n",
    "    res = es.search(index=index_name, body=body)\n",
    "    hits = res[\"hits\"][\"hits\"]\n",
    "    contexts, books_info = [], []\n",
    "\n",
    "    for hit in hits:\n",
    "        book = hit[\"_source\"]\n",
    "        context = (\n",
    "            f\"{book['book_title']} by {book['author_name']}: {book['book_description']}\"\n",
    "        )\n",
    "        contexts.append(context)\n",
    "        books_info.append(book)\n",
    "\n",
    "    return contexts, books_info"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3caa03c",
   "metadata": {},
   "source": [
    "## Implementing the RAG generation component\n",
    "\n",
    "This section sets up the RAG generation pipeline by configuring the OpenAI API key, creating a chat model, defining functions to generate answers from retrieved context, analyze question intent, score books against user preferences, and produce ground truth responses for evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7368d27e-4347-4ba5-8a1d-08500edb37a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Ensure the OpenAI API key is available in the environment\n",
    "\n",
    "\n",
    "if \"OPENAI_API_KEY\" not in os.environ:\n",
    "    os.environ[\"OPENAI_API_KEY\"] = getpass(\"OPENAI_API_KEY: \")\n",
    "\n",
    "\n",
    "# Retrieve the API key from the environment\n",
    "\n",
    "API_KEY = os.environ[\"OPENAI_API_KEY\"]\n",
    "\n",
    "# Create a ChatOpenAI client for generating answers\n",
    "\n",
    "chat_llm = ChatOpenAI(model=\"gpt-4o\", temperature=0.1, api_key=API_KEY)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6119e008-b231-4f10-aff9-d5e5acc1480a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uses the retrieved contexts to generate a book recommendation via the LLM\n",
    "\n",
    "\n",
    "def generate_answer(question, contexts):\n",
    "    context_text = \"\\n\\n\".join(contexts)\n",
    "\n",
    "    prompt = f\"\"\"You are a helpful assistant that recommends books.\n",
    "Use only the information from the context below to answer the question.\n",
    "Do not include any books, authors, or details that are not explicitly present in the context.\n",
    "\n",
    "Repeat the exact book title and author from the context in your answer.\n",
    "\n",
    "Context:\n",
    "{context_text}\n",
    "\n",
    "Question:\n",
    "{question}\n",
    "\n",
    "Answer:\"\"\"\n",
    "    response = chat_llm.invoke(prompt)\n",
    "\n",
    "    return response.content.strip()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f9312c85-dafb-476b-809c-82f5042a1ed6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Detects genres, quality preferences, and author focus from the user’s question\n",
    "\n",
    "\n",
    "def analyze_question_intent(question):\n",
    "    question_lower = question.lower()\n",
    "\n",
    "    intent_patterns = {\n",
    "        \"genre_specific\": {\n",
    "            \"science fiction\": [\n",
    "                \"science fiction\",\n",
    "                \"sci-fi\",\n",
    "                \"space\",\n",
    "                \"future\",\n",
    "                \"alien\",\n",
    "                \"technology\",\n",
    "            ],\n",
    "            \"fantasy\": [\"fantasy\", \"magic\", \"dragon\", \"wizard\", \"medieval\", \"kingdom\"],\n",
    "            \"mystery\": [\n",
    "                \"mystery\",\n",
    "                \"detective\",\n",
    "                \"crime\",\n",
    "                \"murder\",\n",
    "                \"investigation\",\n",
    "                \"thriller\",\n",
    "            ],\n",
    "            \"romance\": [\"romance\", \"love\", \"relationship\", \"romantic\"],\n",
    "            \"horror\": [\"horror\", \"scary\", \"ghost\", \"supernatural\", \"fear\"],\n",
    "            \"historical\": [\"historical\", \"history\", \"war\", \"period\", \"ancient\"],\n",
    "            \"biography\": [\"biography\", \"memoir\", \"life story\", \"autobiography\"],\n",
    "            \"non-fiction\": [\n",
    "                \"non-fiction\",\n",
    "                \"nonfiction\",\n",
    "                \"factual\",\n",
    "                \"real\",\n",
    "                \"educational\",\n",
    "            ],\n",
    "        },\n",
    "        \"quality_indicators\": {\n",
    "            \"high_rating\": [\n",
    "                \"high rating\",\n",
    "                \"highly rated\",\n",
    "                \"best rated\",\n",
    "                \"top rated\",\n",
    "                \"excellent\",\n",
    "            ],\n",
    "            \"popular\": [\"popular\", \"bestseller\", \"well-known\", \"famous\", \"acclaimed\"],\n",
    "            \"award_winning\": [\"award\", \"prize\", \"winner\", \"acclaimed\", \"celebrated\"],\n",
    "            \"classic\": [\"classic\", \"timeless\", \"masterpiece\", \"legendary\"],\n",
    "            \"recent\": [\"recent\", \"new\", \"latest\", \"modern\", \"contemporary\"],\n",
    "        },\n",
    "        \"author_focus\": [\"author\", \"writer\", \"by\", \"written by\"],\n",
    "    }\n",
    "\n",
    "    detected_genres = []\n",
    "    for genre, keywords in intent_patterns[\"genre_specific\"].items():\n",
    "        if any(keyword in question_lower for keyword in keywords):\n",
    "            detected_genres.append(genre)\n",
    "\n",
    "    quality_preferences = []\n",
    "    for quality_type, keywords in intent_patterns[\"quality_indicators\"].items():\n",
    "        if any(keyword in question_lower for keyword in keywords):\n",
    "            quality_preferences.append(quality_type)\n",
    "\n",
    "    author_focused = any(\n",
    "        keyword in question_lower for keyword in intent_patterns[\"author_focus\"]\n",
    "    )\n",
    "\n",
    "    return {\n",
    "        \"genres\": detected_genres,\n",
    "        \"quality_preferences\": quality_preferences,\n",
    "        \"author_focused\": author_focused,\n",
    "        \"question_lower\": question_lower,\n",
    "        \"genre_keywords\": intent_patterns[\"genre_specific\"],\n",
    "    }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "08baf9af-1b73-4897-8113-e7ff3b7e7594",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Scores books based on intent matching, rating, and content relevance.\n",
    "\n",
    "\n",
    "def calculate_book_score(book, intent_data):\n",
    "    score = 0\n",
    "    reasons = []\n",
    "\n",
    "    rating = float(book.get(\"rating_score\", 0))\n",
    "    score += rating * 10\n",
    "\n",
    "    book_title = book.get(\"book_title\", \"\").lower()\n",
    "    book_desc = book.get(\"book_description\", \"\").lower()\n",
    "    author_name = book.get(\"author_name\", \"\")\n",
    "\n",
    "    for genre in intent_data[\"genres\"]:\n",
    "        genre_keywords = intent_data[\"genre_keywords\"][genre]\n",
    "        if any(\n",
    "            keyword in book_desc or keyword in book_title for keyword in genre_keywords\n",
    "        ):\n",
    "            score += 30\n",
    "            reasons.append(f\"matches {genre} genre\")\n",
    "            break\n",
    "\n",
    "    if \"high_rating\" in intent_data[\"quality_preferences\"] and rating >= 4.0:\n",
    "        score += 20\n",
    "        reasons.append(\"high rating\")\n",
    "\n",
    "    if \"popular\" in intent_data[\"quality_preferences\"] and len(book_desc) > 200:\n",
    "        score += 15\n",
    "        reasons.append(\"comprehensive description suggests popularity\")\n",
    "\n",
    "    if intent_data[\"author_focused\"] and author_name:\n",
    "        score += 10\n",
    "        reasons.append(\"has clear author attribution\")\n",
    "\n",
    "    stop_words = [\n",
    "        \"a\",\n",
    "        \"an\",\n",
    "        \"the\",\n",
    "        \"is\",\n",
    "        \"are\",\n",
    "        \"what\",\n",
    "        \"can\",\n",
    "        \"you\",\n",
    "        \"me\",\n",
    "        \"i\",\n",
    "        \"book\",\n",
    "        \"books\",\n",
    "    ]\n",
    "    question_words = [\n",
    "        word for word in intent_data[\"question_lower\"].split() if word not in stop_words\n",
    "    ]\n",
    "\n",
    "    desc_matches = sum(1 for word in question_words if word in book_desc)\n",
    "    if desc_matches > 0:\n",
    "        score += desc_matches * 5\n",
    "        reasons.append(f\"description matches {desc_matches} key terms\")\n",
    "\n",
    "    return {\"book\": book, \"score\": score, \"reasons\": reasons, \"rating\": rating}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "438e24fb-5f54-4b82-905b-833186283392",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creates a textual recommendation from the top-scoring books.\n",
    "\n",
    "\n",
    "def generate_ground_truth_response(top_books, intent_data):\n",
    "    if not top_books:\n",
    "        return \"No relevant books found.\"\n",
    "\n",
    "    top_book = top_books[0]\n",
    "    book = top_book[\"book\"]\n",
    "    title = book[\"book_title\"]\n",
    "    author = book[\"author_name\"]\n",
    "    rating = top_book[\"rating\"]\n",
    "\n",
    "    detected_genres = intent_data[\"genres\"]\n",
    "    quality_preferences = intent_data[\"quality_preferences\"]\n",
    "    author_focused = intent_data[\"author_focused\"]\n",
    "\n",
    "    if detected_genres:\n",
    "        genre = detected_genres[0]\n",
    "        if \"high_rating\" in quality_preferences or rating >= 4.0:\n",
    "            response = f\"For a highly-rated {genre} book, I recommend '{title}' by {author} (rating: {rating:.1f}).\"\n",
    "        else:\n",
    "            response = f\"A good {genre} book from the available options is '{title}' by {author}.\"\n",
    "\n",
    "    elif \"high_rating\" in quality_preferences:\n",
    "        response = f\"Among the highest-rated books available, '{title}' by {author} stands out with a {rating:.1f} rating.\"\n",
    "\n",
    "    elif \"popular\" in quality_preferences:\n",
    "        response = f\"'{title}' by {author} appears to be a popular choice based on the comprehensive information available.\"\n",
    "\n",
    "    elif author_focused:\n",
    "        response = f\"I recommend '{title}' by the author {author}.\"\n",
    "\n",
    "    else:\n",
    "        if rating >= 4.0:\n",
    "            response = f\"I recommend '{title}' by {author}, which has a strong rating of {rating:.1f}.\"\n",
    "        else:\n",
    "            response = f\"Based on the available books, '{title}' by {author} would be a good choice.\"\n",
    "\n",
    "    if len(top_books) > 1 and top_books[1][\"score\"] > top_books[0][\"score\"] * 0.8:\n",
    "        second_book = top_books[1][\"book\"]\n",
    "        response += f\" You might also consider '{second_book['book_title']}' by {second_book['author_name']}.\"\n",
    "\n",
    "    return response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a5f736d-3896-4ec9-86fe-a6afcd7a47b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Orchestrates scoring and response generation for ground truth creation.\n",
    "\n",
    "\n",
    "def create_ground_truth(question, books_info):\n",
    "    if not books_info:\n",
    "        return \"No relevant books found.\"\n",
    "\n",
    "    intent_data = analyze_question_intent(question)\n",
    "\n",
    "    book_scores = []\n",
    "    for book in books_info:\n",
    "        scored_book = calculate_book_score(book, intent_data)\n",
    "        book_scores.append(scored_book)\n",
    "\n",
    "    book_scores.sort(key=lambda x: x[\"score\"], reverse=True)\n",
    "\n",
    "    return generate_ground_truth_response(book_scores, intent_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "beeafb3f",
   "metadata": {},
   "source": [
    "## Running the demo\n",
    "\n",
    "This section runs the RAG demo by asking sample questions, retrieving context, generating answers, creating ground truths, building an evaluation dataset, running Ragas metrics, and printing and saving the evaluation results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "114a1635-6f8d-4597-a0e5-827b5e21e0f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Executes the full demo, collects results, and runs a Ragas evaluation.\n",
    "\n",
    "\n",
    "def run_ragas_demo():\n",
    "    print(\"🚀 Demo:\\n\")\n",
    "\n",
    "    demo_questions = [\n",
    "        \"What's a good science fiction book with high ratings?\",\n",
    "        \"Can you suggest a fantasy book by a popular author?\",\n",
    "        \"What's a highly rated mystery novel?\",\n",
    "        \"Recommend a book with good reviews\",\n",
    "    ]\n",
    "\n",
    "    questions, contexts_list, answers, ground_truths = [], [], [], []\n",
    "\n",
    "    for i, question in enumerate(demo_questions, 1):\n",
    "        print(f\"\\n📚 Question {i}: {question}\")\n",
    "\n",
    "        try:\n",
    "            contexts, books_info = vector_search(question, top_k=3)\n",
    "            if not contexts:\n",
    "                print(f\"No contexts found for question {i}\")\n",
    "                continue\n",
    "\n",
    "            answer = generate_answer(question, contexts)\n",
    "            print(f\"Answer: {answer[:100]}...\")\n",
    "            ground_truth = create_ground_truth(question, books_info)\n",
    "            print(f\"Ground Truth: {ground_truth}\")\n",
    "            questions.append(question)\n",
    "            contexts_list.append(contexts)\n",
    "            answers.append(answer)\n",
    "            ground_truths.append(ground_truth)\n",
    "\n",
    "        except Exception as e:\n",
    "            print(f\"Error processing question {i}: {e}\")\n",
    "            continue\n",
    "\n",
    "    if not questions:\n",
    "        print(\"\\nNo valid Q&A pairs generated.\")\n",
    "        return None\n",
    "\n",
    "    eval_dataset = Dataset.from_dict(\n",
    "        {\n",
    "            \"question\": questions,\n",
    "            \"contexts\": contexts_list,\n",
    "            \"answer\": answers,\n",
    "            \"ground_truth\": ground_truths,\n",
    "        }\n",
    "    )\n",
    "\n",
    "    print(\"\\n✨ Running Ragas evaluation...\")\n",
    "    try:\n",
    "        result = evaluate(\n",
    "            dataset=eval_dataset,\n",
    "            metrics=[context_precision, faithfulness, context_recall],\n",
    "            llm=chat_llm,\n",
    "            embeddings=None,\n",
    "        )\n",
    "\n",
    "        df = result.to_pandas()\n",
    "\n",
    "        print(\"\\n✨ Ragas Evaluation Results:\")\n",
    "        print(df)\n",
    "\n",
    "        print(\"\\✨ Averages:\")\n",
    "\n",
    "        for metric, value in df.mean(numeric_only=True).items():\n",
    "            print(f\"{metric}: {value:.3f}\")\n",
    "\n",
    "        df.to_csv(\"ragas_evaluation.csv\", index=False)\n",
    "        print(\"\\nResults saved to 'ragas_evaluation.csv'\")\n",
    "        return result\n",
    "\n",
    "    except Exception as e:\n",
    "        print(f\"Ragas evaluation failed: {e}\")\n",
    "        return None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "db4685c5-7cc0-4e2a-875f-bc233d6d4591",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🚀 Demo:\n",
      "\n",
      "\n",
      "📚 Question 1: What's a good science fiction book with high ratings?\n",
      "Answer: \"Light of the Jedi\" by Charles Soule...\n",
      "Ground Truth: For a highly-rated science fiction book, I recommend 'Light of the Jedi' by Charles Soule (rating: 4.2). You might also consider 'Legion' by Dan Abnett.\n",
      "\n",
      "📚 Question 2: Can you suggest a fantasy book by a popular author?\n",
      "Answer: Based on the context provided, I recommend \"The Book of Time\" by Guillaume Prévost....\n",
      "Ground Truth: A good fantasy book from the available options is 'Our Pet' by S.M. Matthews. You might also consider 'The Book of Time' by Guillaume Prévost.\n",
      "\n",
      "📚 Question 3: What's a highly rated mystery novel?\n",
      "Answer: Human Nature by Jonathan Green...\n",
      "Ground Truth: For a highly-rated mystery book, I recommend 'Human Nature' by Jonathan Green (rating: 3.6). You might also consider 'Lamb: The Gospel According to Biff, Christ's Childhood Pal' by Christopher Moore.\n",
      "\n",
      "📚 Question 4: Recommend a book with good reviews\n",
      "Answer: The Island of Doctor Moreau by H.G. Wells...\n",
      "Ground Truth: Based on the available books, 'The Island of Doctor Moreau' by H.G. Wells would be a good choice. You might also consider 'Savage Drift' by Emmy Laybourne.\n",
      "\n",
      "✨ Running Ragas evaluation...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "80d98ec6b77248a3b1762429b3b2a0c8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Evaluating:   0%|          | 0/12 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "✨ Ragas Evaluation Results:\n",
      "                                          user_input  \\\n",
      "0  What's a good science fiction book with high r...   \n",
      "1  Can you suggest a fantasy book by a popular au...   \n",
      "2               What's a highly rated mystery novel?   \n",
      "3                 Recommend a book with good reviews   \n",
      "\n",
      "                                  retrieved_contexts  \\\n",
      "0  [Light of the Jedi by Charles Soule: Two hundr...   \n",
      "1  [Alien Warrior's Mate by Vi Voxley: He's damn ...   \n",
      "2  [The Island of Doctor Moreau by H.G. Wells: Ra...   \n",
      "3  [The Island of Doctor Moreau by H.G. Wells: Ra...   \n",
      "\n",
      "                                            response  \\\n",
      "0               \"Light of the Jedi\" by Charles Soule   \n",
      "1  Based on the context provided, I recommend \"Th...   \n",
      "2                     Human Nature by Jonathan Green   \n",
      "3          The Island of Doctor Moreau by H.G. Wells   \n",
      "\n",
      "                                           reference  context_precision  \\\n",
      "0  For a highly-rated science fiction book, I rec...                0.0   \n",
      "1  A good fantasy book from the available options...                0.5   \n",
      "2  For a highly-rated mystery book, I recommend '...                0.0   \n",
      "3  Based on the available books, 'The Island of D...                1.0   \n",
      "\n",
      "   faithfulness  context_recall  \n",
      "0      1.000000             0.0  \n",
      "1      1.000000             1.0  \n",
      "2      0.666667             0.0  \n",
      "3      0.666667             1.0  \n",
      "\\✨ Averages:\n",
      "context_precision: 0.375\n",
      "faithfulness: 0.833\n",
      "context_recall: 0.500\n",
      "\n",
      "Results saved to 'ragas_evaluation.csv'\n",
      "\n",
      "🎉 Demo completed successfully!\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    results = run_ragas_demo()\n",
    "    if results:\n",
    "        print(\"\\n🎉 Demo completed successfully!\")\n",
    "    else:\n",
    "        print(\"\\nDemo completed with issues.\")\n",
    "\n",
    "except Exception as e:\n",
    "    print(f\"\\nError during demo: {e}\")\n",
    "    import traceback\n",
    "\n",
    "    traceback.print_exc()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
